Fundamentals of Missing Data in Evaluation

Presentation to MSU Department of Psychology, Program Evaluation Occasional Speaker Series, East Lansing, MI

Steven J. Pierce

Center for Statistical Training and Consulting

2024-12-05

Outline

  • What is missing data?
  • Why do we end up with missing data?
  • Why should we care about missing data?
  • How can we diagnose the missing data issues for a given study?
  • What should we do about missing data?
    • Prevention
    • Diagnosis
    • Treatment
  • Advice

What is missing data?

Missing data (MD) are measurements you want or intended to collect but did not get.[1]

  • Having MD is common in research & evaluation studies.
  • If you do much evaluation work, you will run into MD.

Why do we end up with missing data?

Data collection doesn’t always go according to plan…

Human Factors Other Factors
Participant behavior Equipment failures
Evaluator errors Records/Databases
Partner behavior Unusual Events

Missing Data & Project Lifecycle

WhenMD Plan Study Planning & Design Collect Data Collection Plan->Collect Enter Data Entry Collect->Enter Manage Data Storage & Management Enter->Manage Analyze Data Analysis Manage->Analyze

Why should we care about missing data?

Ethics for Evaluators

Handling missing data well enacts our guiding principles[2]:

AEA logo.

  • Systematic inquiry
  • Competence
  • Integrity

Scientific Activities[3]

There are 3 major scientific activities that can be affected by missing data.

  • Making structured observations of constructs.
  • Using observations to draw inferences about relationships between constructs.
  • Generalizing the results to populations beyond the collected sample.

Consequences for Measurement[3]

  • Availability of constructs
  • Decreased reliability due to increased error variance
  • Bias from poor content coverage
  • Construct validity

Consequences for Internal Validity[1,3]

  • Selection bias
  • Compromised randomization
  • Power and precision
  • Inaccurate model assumptions

Consequences for Generalizability[1,3]

A representative samples is crucial to generalizing to the intended population!

  • Theory development & cumulative knowledge
  • Policy & decision-making

Types of Missingness

  • Item-level
  • Construct
  • Person-period
  • Planned vs unplanned

Mechanisms of Missingness

  • Missing completely at random (MCAR)
  • Missing at random (MAR)
  • Missing not at random (MNAR)

MCAR

MCAR is when neither observed nor unobserved variables predict which data is missing.

MAR

MAR is when observed values predict which data is missing.

MNAR

MNAR is when unobserved values predict which values are missing.

Diagnosis

Describing the Amount of MD

[3]

  • Numbers & percentages of complete & incomplete cases
  • Number and percentage of missing values for each variable
  • Nature and frequency of missing data patterns

Predictors of Attrition & Missingness in Longitudinal Studies

  • Study arm: Compare retention rates
  • Study site (in multisite studies)
  • Baseline/pretest values of outcome variables may predict who drops out or has missing values
  • Other covariates (demographics, site, )

Treatment

Prevention

An ounce of prevention is better than a pound of cure

[4],[5]

Advice

  • Collaborate with a statistician!

Practical Options

  • Item-level missingness in scale scores[6,7]

References

1. Fernández-García, M. P., Vallejo-Seco, G., Livácic-Rojas, P., & Tuero-Herrero, E. (2018). The (ir)responsibility of (under)estimating missing data. Frontiers in Psychology, 9(556). https://doi.org/10.3389/fpsyg.2018.00556
2. American Evaluation Association. (2018). Guiding principles for evaluators [Web Page]. Author. https://www.eval.org/About/Guiding-Principles
3. McKnight, P. E., McKnight, K. M., Sidani, S., & Figueredo, A. J. (2007). Missing data: A gentle introduction. Guilford Press.
4. Leeuw, E. D. de. (2001). Reducing missing data in surveys: An overivew of methods. Quality & Quantity, 35(2), 147–160. https://doi.org/10.1023/A:1010395805406
5. Wisniewski, S. R., Leon, A. C., Otto, M. W., & Trivedi, M. H. (2006). Prevention of missing data in clinical research studies. Biological Psychiatry, 59, 997–1000. https://doi.org/10.1016/j.biopsych.2006.01.017
6. Graham, J. W. (2009). Missing data analysis: Making it work in the real world. Annual Review of Psychology, 60, 549–576. https://doi.org/10.1146/annurev.psych.58.110405.085530
7. Newman, D. A. (2014). Missing data: Five practical guidelines. Organizational Research Methods, 17(4), 372–411. https://doi.org/10.1177/1094428114548590